This PR contains the second part of #165, including the actual mechanics of log forwarding in Pebble. It builds upon #209 and #252. This includes an abstract logClient interface but doesn't include the specific Loki/syslog implementations - this will come in later PRs.

This is a modification of #218, designed to fix fundamental issues identified with that PR.

Current design

For each log target, there is a single "gatherer", which receives logs from a bunch of services and writes them to a "client". The gatherer runs a control loop in a separate goroutine, which waits to receive logs on a channel, and writes these to the client.

For each service, the gatherer spawns a "log puller". Each puller runs in a separate goroutine, pulling logs from an iterator on the service's ringbuffer. Pulled logs are sent to the gatherer's control loop on the shared channel.

The logClient interface represents a client to a specific type of backend. In future PRs, we will add lokiClient and syslogClient types which implement logClient.

logClient includes two methods

type logClient interface {
	Write(context.Context, servicelog.Entry) error
	Flush(context.Context) error
}

Client implementations have some freedom about the semantics of these methods.

• For a buffering client (e.g. HTTP), Write could add the log to the client's internal buffer, while Flush would prepare and send an HTTP request with the buffered logs.
• For a non-buffering client (TCP?), Write could serialise the log directly to the open connection, while Flush would be a no-op.

Teardown

Gracefully stopping a log gatherer is a complex process with multiple steps.

• At the instant we attempt to stop the gatherer, it may be in the middle of flushing its client. So, wait a small amount of time for the client to finish flushing, but cancel the flush if this takes too long.
• The service may have emitted some final logs on shutdown. Give each puller some time to pull the final logs from its iterator - but again, force kill it if this is taking too long.
• Once the pullers are all finished, we must have received all the logs we're gonna get, so we can safely shut down the main loop.
• Do a final flush of the client to send off any remaining buffered logs.

All this logic is encapsulated in the gatherer.stop() method.

QA

I've included some sample implementations of logClient here. They just print the logs to stdout. These can be used to verify that the log forwarding mechanics work properly.

Create a simple logging service, e.g.

#!/bin/bash
while true; do
  echo "Hello"
  sleep 1
done

and a simple plan using this service

services:
  svc1: &logsvc
    command: /home/jb/git/canonical/pebble/logfwd-impl2/pebble/logsvc
    startup: enabled
    override: merge
  svc2: *logsvc

log-targets:
  tgt1:
    override: merge
    services: [all]
    type: loki
    location: unnecessary

Add the fake.go file to the repo.

Comment out the following line

return &nonBufferingClient{}, nil          // unbuffered
return &bufferingClient{}, nil             // unlimited-size buffer, will flush on timeout only
return &bufferingClient{threshold: 3}, nil // buffer with max size: 3 logs

You might also want to change the gatherer's tick period:

Run Pebble with

go run ./cmd/pebble run

and verify the logs are printing to stdout.

JUJU-3776